Field of the Invention
The present invention relates to an image processing apparatus and an image processing method.
Description of the Related Art
For example, an array of microlenses (MLs), which are arranged in the proportion of one to a plurality of pixels, is placed on a front surface of an image pickup element to acquire not only a two-dimensional intensity distribution of light but also information on a direction of incidence of a ray that enters the image pickup element, with the result that three-dimensional information on an object space may be obtained.
A camera (image pickup apparatus) capable of obtaining such three-dimensional information on the object space is called a “light field camera”. Moreover, the three-dimensional information on the object space is called “light field data (LF data)”. Acquisition of the LF data and image reconstruction processing after the photography enable image processing called “refocusing” such as changing a focus position of an image, changing a photographing view point, and adjusting a depth of field with a focus position at the time of acquiring the LF data being a center.
In the light field camera as described above, a plenoptic system has widely been known. In the plenoptic system, a plurality of photoelectric transducers (photodiodes) for photography are two-dimensionally arranged under the microlens array, and a focus lens included in an optical system serves as an exit pupil of each of the microlenses. In the image pickup apparatus having the above-mentioned configuration, it is known that a plurality of pieces of ray information from an object may be obtained by the photoelectric transducers located under the microlens. A two-dimensional image formed only of pixels at the same position in pixel groups located under the microlenses by using such ray information has a parallax with respect to a two-dimensional image formed only of pixels at a different position. The two-dimensional images having such parallaxes are combined to virtually move a focus plane of a recorded image.
The number of image pickup pixels of the above-mentioned plenoptic camera is expressed by the following expression (1):LF data=N×M  (1),where N represents the number of MLs of the image pickup element, and M represents the number of divisions of an image to be divided by one microlens.
Note that, the number of pieces of final output pixel data to be combined based on such LF data is the same as N, which is the number of microlenses.
As described above, the number of pieces of LF data is increased in accordance with the number of divided photoelectric transducers under the microlens, and in an image pickup apparatus having the refocusing function, when various kinds of signal processing are performed on the number of pieces of output pixel data, which is the final output, a processing load is increased. Further, when the image pickup pixel data is captured in various kinds of image signal processing units, a data storing unit having a large capacity is required. As a result, power consumption is increased with an increase in load of data processing, and further, the cost is increased due to an increase in capacity of the data storing unit with an increase in amount of data to be processed.
In order to address the above-mentioned problems, in Japanese Patent Application Laid-Open No. 2013-247590, there is disclosed a technology in which it is determined whether or not there is a possibility of performing refocusing depending on a distance at which an object is located, all pixels under the microlens are recorded in a case where there is a possibility of performing the refocusing, and the pixels are added and recorded in a case where there is no possibility of performing the refocusing.